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SELECTION  OF  AIR  TRAFFIC  CONTROLLERS  FOR  AUTOMATED  SYSTEMS: 
APPLICATIONS  FROM  TODAY’S  RESEARCH 


INTRODUCTION 

Over  the  next  two  decades,  the  Federal  Aviation  Administration’s  (FAA)  National 
Airspace  System  (NAS)  Plan  for  new  automated  air  traffic  control  systems  will 
radically  change  the  job  of  Air  Traffic  Control  Specialist  (ATCS).  While  the 
introduction  of  automation  is  not  new  to  the  occupation,  the  NAS  Flan  calls  for 
automating  more  critical  job  tasks  than  could  have  been  previously  supported  by 
technology.  The  challenge  of  selecting  the  controller*  to  operate  these  future 
automated  systems  is  currently  facing  the  FAA. 

The  first  step  in  designing  a  selection  system  for  current  or  future  jobs  is  an 
analysis  of  the  tasks  required  to  perform  the  job.  In  approaching  such  analysis,  a 
would-be  designer  of  a  selection  system  unfamiliar  with  air  traffic  control  faces  a 
challenge.  Hopkin  (1)  has  captured  first  impressions  of  the  job  in  the  following 
comments: 


"To  a  naive  observer,  the  workspace  of  controllers  seems  complex  and  well 
nigh  meaningless.  It  is  not  intuitively  and  immediately  obvious  what  any  of 
the  data  mean,  what  the  tasks  are,  how  they  are  done,  or  what  would 
constitute  successful  performance." 


Thus,  the  first  step  toward  a  successful  selection  system  involves  analysis  of  tasks 
not  immediately  obvious  and  perhaps  not  easily  measured. 

By  definition,  a  controller  is  tasked  with  promoting  the  safe,  orderly  and  expeditious 
flow  of  air  traffic.  This  is  accomplished  through  accurate,  effective  application  of 
rules  and  procedures  in  a  real-time,  dynamic  environment.  The  current  ATCS’s  job 
consists  of  a  complex  set  of  tasks  that  demand  a  high  degree  of  skill  and  active 
application  of  certain  cognitive  abilities,  such  as  spatial  perception,  information 
processing,  reasoning,  and  decision  making.  As  an  increasing  number  of  ATC  tasks 
become  automated,  the  controller’s  participation  may  gradually  evolve  from  an  active 
manager  to  a  more  passive  system  monitor. 


*The  terms  ATCS  and  controller  are  used  interchangeably  throughout  this  paper. 
ATCS  is  the  formal  job  title,  whereas  "controller"  is  the  more  familiar,  informal  title. 
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The  current  work  force  has  been  rigorously  selected  through  a  two  stage  selection 
system.  The  first  stage  consists  of  a  paper-and-pencil  aptitude  test  battery.  The 
second  stage  is  a  performance-based  measure  consisting  of  a  condensed  training¬ 
testing  ATC  task  sample  in  a  nonradar  environment.  As  the  job  itself  changes, 
questions  arise  as  to  how  long  a  selection  system  based  upon  the  current  selection 
criteria  will  be  effective,  and  what  factors  will  be  important  in  designing  the 
replacement. 

To  address  these  questions,  the  job  of  the  ATCS  in  its  current  form  will  be  reviewed 
and  contrasted  to  projected  changes  in  job  tasks  which  will  result  from 
implementation  of  certain  components  of  the  NAS  Plan;  specifically  the  Initial  Sector 
Suite  System  (ISSS)  and  the  Automated  En  Route  Air  Traffic  Control  (AERA) 
software  enhancements  of  the  Advanced  Automation  System  (AAS).  The  current 
selection  system  will  then  be  examined  to  determine  the  extent  to  which  the  current 
selection  criteria  might  predict  success  in  future  ATC  jobs  as  automation  causes  the 
tasks  to  evolve. 


AUTOMATION  AND  AIR  TRAFFIC  CONTROL  SERVICES 


Over  the  50-year  history  of  the  ATC  system’s  evolution,  a  mixture  of  equipment  of 
varying  technological  generations  and  types  was  used.  Computer  automation 
techniques  were  first  applied  to  flight  data  processing  systems  utilizing  UNIVAC  and 
IBM  computers  during  the  early  1960s.  The  systems  were  installed  in  six 
northeastern  Air  Route  Traffic  Control  Centers  (ARTCCs).  Through  this  effort  the 
FAA  demonstrated  that  computers  could  be  used  to  improve  aviation  safety  and 
increase  the  controller’s  productivity.  In  1968  an  integrated  plan  to  automate  ATC 
functions  was  approved.  This  plan  was  NAS  Stage  A.  By  1981  the  NAS  Plan  for 
Facilities,  Equipment  and  Associated  Development  was  issued  to  define  an  orderly 
rational  evolution  of  the  system  as  a  whole.  The  plan  has  been  updated  annually. 
The  most  recent  version  is  the  NAS  Plan:  Facilities,  Equipment  and  Associated 
Development,  and  other  Capital  Needs,  which  was  issued  in  September  1989  (2). 

The  NAS  Plan  established  three  principal  objectives  for  the  en  route  and  terminal 
ATC  systems.  The  first  was  consolidation  of  more  than  200  facilities  existing  today 
into  less  than  30  by  the  year  2000.  The  second  objective  was  to  install  common 
modular  computers,  software,  and  controller  work  stations  (i.e.,  "Sector  Suites").  To 
meet  these  two  objectives,  the  en  route  and  terminal  radar  approach  control  facilities 
will  be  consolidated  into  area  control  facilities  (ACF’s)  with  essentially  identical 
equipment,  including  the  HOST  computer  and  the  Initial  Sector  Suite  System  (ISSS). 
The  third  objective  was  the  improvement  of  safety,  fuel  efficiency  and  productivity 
through  higher  levels  of  automation.  The  NAS  Plan  includes  improvements  such  as 
Mode  S  data  link,  ASR-9  and  advanced  Doppler  weather  radar,  navigation  by  Global 
Positioning  System  (GPS)  satellites,  and  advanced  stages  of  automation  software, 
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such  as  Automated  En  Route  Air  Traffic  Control  (AERA).  Other  advances  in  the  air 
traffic  system  that  will  affect  the  controllers’  functions  include  point-to-point  navigation 
and  direct  routing  from  point  of  departure  to  destination. 

The  NAS  Plan  provides  for  a  staged  delivery  of  the  various  equipment  and  software 
developments.  The  HOST  computers  have  been  installed  and  facilities  await 
delivery  of  the  first  Sector  Suites  beginning  in  the  mid-1990s. 


NAS  PLAN  AND  THE  AIR  TRAFFIC  CONTROL  SPECIALIST 

Two  automation  advancements  in  the  NAS  Plan  will  particularly  affect  the 
environment  and  job  tasks  of  the  en  route  controller:  Sector  Suites  and  the 
implementation  of  AERA. 

The  Sector  Suites  provide  an  entirely  new  controller  workstation.  Briefly  described, 
each  Sector  Suite  is  a  collection  of  one  to  four  common  consoles,  each  of  which 
contain  data  entry  and  display  equipment.  The  Sector  Suite  consoles  will  provide 
information  to  the  controller  on  multiple  display  screens.  A  common  console  will 
consist  of  a  main  and  an  auxiliary  display,  voice  switching,  and  control  panels. 
Display  screens  will  have  multi-color  capability.  Data  entry  devices  will  include  a 
keyboard  and  trackball.  Information  presented  will  include  a  situation  display  of  the 
air  traffic  and  weather,  alphanumeric  flight  and  weather  data,  as  well  as  other 
pertinent  information,  such  as  Notices  to  Airmen  (NOTAM’s).  The  most  substantial 
change  to  the  controller’s  environment  presented  by  the  Sector  Suites  is  the 
replacement  of  paper  flight  progress  strips  with  an  electronic  display  of  flight  data 
(3). 

AERA  will  also  substantially  affect  the  en  route  controller’s  job  tasks  by  developing  a 
software  system,  which  will  enhance  the  earlier  ATC  hardware  and  software  system. 
AERA  will  support  the  ATCS  by  predicting  and  resolving  problems  along  an 
aircraft’s  route  of  flight,  and  by  planning  and  maintaining  traffic  flow  and  aircraft 
separation. 

The  NAS  Plan  proposes  three  stages  of  AERA  software  enhancements:  AERA  1, 
AERA  2,  and  AERA  3.  Under  AERA  1,  the  computer  software  will  detect  certain 
common  problems,  such  as  conflicts  between  aircraft,  violations  of  protected 
airspace,  and  non-adherence  to  ATC-imposed  traffic  flow  restrictions.  AERA  2  will 
then  provide  enhanced  assistance  to  controllers  through  proposed  resolution  of 
detected  problems.  Several  resolutions  for  a  problem  will  be  generated.  The 
highest  ranked  resolution  will  be  presented  to  the  controller  currently  controlling  the 
aircraft  affected  by  the  resolution.  The  controller  will  assess  that  resolution,  examine 
others  if  necessary,  and  determine  which  resolution  to  choose,  depending  on  his  or 
her  requirements  for  the  airspace.  This  activity  will  be  complicated  by  the  increased 
use  of  direct  routings  instead  of  currently  established  jetways.  For  aircraft  equipped 
with  data  link,  clearances  will  be  transmitted  by  the  computer,  once  approved  by  the 
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controller.  Clearances  for  aircraft  without  data  link  will  be  delivered  by  the  controller 
through  a  voice  channel.  Certain  coordination  functions  between  controllers  will  also 
be  automated.  The  controller  remains  responsible  for  all  control  actions  with  AERA 
2.  However,  with  AERA  3,  automated  decision-making  and  control  will  be 
introduced  with  a  certain  degree  of  autonomy  for  the  computer  to  detect  and  resolve 
ATC  problems  without  human  intervention. 

The  NAS  Plan  establishes  a  clear  path  toward  increased  automation  of  air  traffic 
control.  If  the  current  responsibility  of  the  controller  is  to  ensure  the  separation  cf 
aircraft  as  well  as  the  safe,  expeditious  flow  of  traffic,  then  by  the  time  AERA  3  is 
imDlemented,  much  of  that  function  could  shift  to  the  computer.  To  design  a 
selection  system  for  the  evolving  job  of  the  controller  or  to  assess  aspects  of  the 
current  selection  system  that  might  have  utility  for  the  future,  it  is  important  to 
understand  the  job  tasks  of  the  controller  and  the  changes  that  will  be  made  in 
those  tasks  with  increased  automation.  The  following  section  examines  the  current 
functions  of  an  ATCS  and  reviews  studies  analyzing  the  impact  of  Initial  Sector 
Suite  System  and  Automated  En  Route  Air  Traffic  Control  on  those  functions. 


ATC  JOB  TASK  ANALYSES 

A  job  task  analysis  (JTA)  provides  a  detailed  itemization  of  the  tasks  required  by  a 
specific  job.  Such  a  detailed  analysis  can  help  us  systematically  map  changes  in 
job  functions  with  each  phase  of  the  NAS  Plan.  From  there  we  can  begin  to  assess 
the  knowledge,  skills,  and  abilities  required  of  the  human  operator.  In  addition,  the 
JTA  can  help  us  understand  what  we  are  accomplishing  with  current  selection 
systems. 

In  preparation  for  the  transition  to  the  Advanced  Automation  System  (AAS)  under 
the  NAS  Plan,  Computer  Technology  Associates,  Inc.  (CTA)  was  commissioned  to 
conduct  a  job  task  analysis  in  today’s  ATC  environment,  as  well  as  for  the  planned 
automation  capabilities.  The  result  of  this  effort  was  a  multi-volume  documentation 
of  Operations  Concepts  for  ATC  personnel  in  operational  environments  at  different 
stages  of  AAS  evolution.  The  focus  was  placed  on  the  interaction  between  the 
controller  and  the  automated  system  during  operational  tasks  as  described  in 
Volume  l  (4). 

Data  were  collected  through  field  observations,  interviews  conducted  in  a  large 
cross-section  of  ATC  facilities  and  through  regular  involvement  of  a  controller 
validation  team.  Thus,  the  JTA  was  completed  and  validated  with  substantial  input 
from  subject  matter  experts  (i.e.,  ATCSs). 

The  CTA  methodology  derived  a  task  list  utilizing  a  stimulus-process-response  model 
in  which  a  controller  performs  actions  in  response  to  air  traffic  events.  Therefore, 
the  job  analysis  began  with  identifying  and  defining  the  air  traffic  events  to  which  a 
controller  responds.  After  identifying  the  response,  the  controller’s  action  was  then 
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decomposed  to  the  task  level-the  central  analysis  result.  Within  this  framework,  the 
work  performance  actions  were  classified  into  three  increasingly  detailed  levels: 
activities,  sub-activities,  and  tasks.  Activities  were  the  highest  level  of  description  of 
operational  job  functions.  The  operational  duties  of  a  position  were  typically 
described  with  five  to  nine  activities.  Sub-activities  were  a  more  detailed  statement 
of  the  work  performed,  and  a  task  was  the  most  detailed  unit  of  description. 

Within  the  task  level  specification,  four  task  types  were  identified  as  follows:  1) 
ENTRY  tasks  (i.e,  entry  of  data  into  the  system),  2)  RECEIPT  tasks  (e.g.  receipt  of 
information  from  the  displays),  3)  ANALYTICAL  tasks,  and  4)  VERBAL 
COORDINATION  tasks  (coordination  with  other  individuals  or  the  system). 

Task  characterization  analyses  were  also  performed  that  described  tasks  on  four 
separate  dimensions.  Two  of  these  dimensions  are  of  particular  interest  in 
establishing  a  controller  selection  system.  These  two  task  characterizations  are  1) 
the  cognitive/sensory  attributes  required  for  a  task  and  2)  the  identification  of  task 
performance  requirements. 

For  purposes  of  this  paper,  only  the  tasks  required  of  the  controllers  working  in  Air 
Route  Traffic  Control  Centers  (ARTCCs)-the  en  route  controilers-will  be  reviewed. 
These  are  described  in  Volume  VI  of  the  Operations  Concepts  documentation  (5). 
Six  activities  were  identified  for  the  en  route  controller  in  the  current  ATC 
environment,  as  follows: 

1)  perform  situation  monitoring; 

2)  resolve  aircraft  conflicts; 

3)  manage  air  traffic  sequences; 

4)  route  or  plan  flights; 

5)  assess  weather  impact; 

6)  manage  sector/position  resources. 


Within  the  6  activities,  46  sub-activities  and  a  total  of  348  distinct  tasks  were 
identified.  Clearance  delivery  was  identified  as  a  set  of  tasks  that  was  relevant  to 
many  sub-activities  and  was  thus  defined  as  a  "macro"  performed  as  a  function  of  a 
number  of  tasks. 

The  cognitive/sensory  attributes  required  for  performing  tasks  of  extreme  or  high 
criticality  were  identified.  Fourteen  attributes  were  found  to  be  relevant  to 
performance  of  computer  and  radar  workstation  jobs  associated  with  current  ATC: 
coding,  movement  detection,  spatial  scanning,  filtering,  image/pattern  recognition, 
decoding,  visualization,  short-term  memory,  long-term  memory,  deductive  reasoning, 
inductive  reasoning,  rriathematical/probabilistic  reasoning,  prioritizing,  and  verbal 
filtering.  (Definitions  for  these  attributes  are  included  in  Appendix  A). 

Further  Analysis  of  each  of  161  critical  tasks  revealed  that  analytical  attributes  such 
as  visualization  (42  tasks),  short-term  memory  (30  tasks),  deductive  reasoning  (41 
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tasks),  and  mathematical/probabilistic  reasoning  (32  tasks),  were  ks/  cognitive 
attributes.  Additionally,  message  filtering  (40  tasks)  and  decoding  (57  tasks)  were 
important. 


AFFECT  OF  ISSS  AND  AERA  ON  JOB  TASKS 


The  Initial  Sector  Suite  System 

In  1988,  Computer  Technology  Associates  completed  a  report  of  the  impact  of  the 
Initial  Sector  Suite  System  on  ATC  procedures  and  training  (6).  Expanding  on  a 
prior  report,  the  analysis  provided  an  early  identification  of  current  ATC  procedures 
projected  to  be  affected  by  ISSS.  The  analysis  for  the  report  compared  the 
Operations  Concepts  for  the  current  job,  described  above,  and  the  Operations 
Concepts  prepared  for  ISSS  utilizing  the  same  job  analysis  methodologies  (7).  The 
study  estimated  that  48  of  373  (13%)  job  tasks  of  the  en  route  controller  would  be 
substantially  changed  from  the  current  ATC  system;  237  (64%)  would  be  essentially 
unchanged.  The  remaining  88  tasks  would  be  affected  to  some  degree,  but  not 
substantially  changed. 

CTA  observes  that  the  ISSS  introduces  substantia!  changes  to  the  en  route 
controller’s  operations.  In  general,  the  ISSS  will  provide  the  controller  with  greater 
flexibility  and  capability  in  data  handling.  There  will  be  increased  data  processing 
and  flight  data  will  be  available  more  rapidly.  Entry  of  data  into  the  system  will  be 
increased  over  today’s  requirements. 

While  a  comparison  of  information  displayed  reveals  a  distinct  parallelism  between 
current  sector  workstation  physical  displays  and  written  records  and  the  Sector  Suite 
workstation,  the  manner  in  which  controllers  manipulate  the  data  will  be  significantly 
changed.  The  most  substantial  change  with  ISSS  is  the  conversion  of  paper  flight 
progress  strips  to  electronic  flight  data  entry  and  electronic  display  and  manipulation 
of  the  data.  Today’s  flight  progress  strips  are  manually  inserted  into  plastic  holdere, 
placed  in  sequence  in  a  strip  bay,  marked  with  a  pen,  sorted,  flagged,  and  removed. 
Under  ISSS  all  of  these  actions  will  be  accomplished  electronically.  Computer 
Technology  Associates  notes  that  display  management  will  be  much  more  critical 
with  ISSS  than  with  today’s  system,  because  more  information  will  be  available 
under  ISSS  and  will  need  to  be  displayed  for  most  efficient  use.  CTA  also 
expresses  a  concern  that  electronic  presentation  of  data  may  impact  the  silent 
coordination  between  the  radar  and  radar  associate  ("D-side")  controllers. 

In  summary,  the  48  radically  and  significantly  different  tasks  under  ISSS  pertain  to 
the  controllers’  interaction  with  information  displays  and  the  input  of  control  data  and 
messages.  CTA’s  analysis  of  changes  in  the  cognitive/sensory  attributes  of  these 
tasks  determined  that  "coding"  will  be  substantially  increased  because  of  the 
expanded  availability  of  messages.  CTA  concluded  that  cognitive  and  sensory 
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attributes  required  for  the  more  critical  controller  tasks  in  both  the  current  ATC 
system  and  under  ISSS  would  be  similar.  In  terms  of  performance  requirements, 
the  accuracy  of  data  entry  would  become  critical  and  the  frequency  of  receipt  of 
information  from  the  system  would  increase. 


Automated  En  Route  Air  Traffic  Control  (AERA) 

Researchers  at  MITRE,  in  conjunction  with  teams  of  controllers  as  subject  matter 
experts,  prepared  several  papers  assessing  the  changes  in  the  en  route  controllers’ 
operational  functions  after  implementation  of  AERA.  AERA  1  introduces  primarily 
problem  detection  capability  and  some  decision  aids  for  the  en  route  controller. 
However,  AERA  2  expands  these  capabilities  in  several  important  areas.  These 
areas  include  resolutions  to  problems  detected  by  the  automated  problem  detection 
software,  display  of  proposed  resolutions  to  the  appropriate  controller,  and  finally 
provision  of  additional  aids  to  assist  a  controller  in  decision-making  for  separation  of 
air  traffic  and  coordination. 

Carlson  and  Rhodes  (8)  described  the  changes  in  the  controllers’  work  activities  in 
AERA  2.  In  today’s  environment,  controllers  rely  on  their  own  mental  abilities  to 
detect  and  resolve  problems  occurring  in  their  airspace.  ATCSs  must  recognize 
potential  loss  of  separation  between  aircraft  based  upon  available  data  in  flight 
progress  strips  and  plan  view  displays  (situation  displays)  showing  computer 
processed  radar  returns  of  current  and  recent  past  aircraft  positions,  and  their 
knowledge  about  the  operational  ATC  environment.  Upon  recognizing  a  potential 
problem,  the  ATCS  must  formulate  and  evaluate  resolutions  for  the  situation.  The 
human  being  has  a  finite  capacity  to  process  information.  Limits  include  the 
maximum  number  of  aircraft  an  invididi'''1  can  safely  and  efficiently  control  at  any 
one  time,  as  well  as  the  look-ahead  tir  j  to  project  potential  conflicts.  Introduction 
of  direct  routing  of  aircraft  and  reduced  use  of  established  airways  may,  in  fact, 
overload  a  controller’s  information  processing  capabilities. 

AERA  2  should  expand  the  capacity  of  the  VTC  system  by  automating  the  detection 
of  problems  up  to  20  minutes  prior  to  predicted  loss  of  separation  and  for  an 
aircraft’s  entire  route  of  flight.  The  Automated  Problem  Resolution  (APR)  function 
will  produce  a  number  of  resolutions  based  upon  the  aircraft’s  projected  trajectory, 
geometry  of  the  conflict,  airspace  characteristics,  and  other  aircraft  or  system 
requirements.  This  capability  is  projected  to  improve  detection  and  resolution  of 
problems  because  they  will  be  detected  earlier,  allowing  more  optimal  resolutions  to 
be  considered. 

The  affect  of  AERA  2  on  the  controller’s  job  has  been  described  by  Carlson  and 
Rhodes  (8)  as  shifting  the  emphasis  away  from  tactical  decision-making  to  more 
strategic  management  of  the  ATC  environment.  They  suggest  that  many  routine 
tasks  will  be  automated,  and  more  complex  tasks  will  become  easier  because  of  the 
availability  of  computer  proposed  resolutions,  suggesting  that  controllers  will  rely  less 
on  their  own  mental  abilities  to  monitor  flight  data  and  identify  situations  requiring 


control  actions.  The  focus  of  the  controller's  attention  will  shift  from  the  flight  data 
and  situation  display  to  the  problem  alert  and  resolution  displays.  Finally,  they 
suggest  that  because  problems  wil*  be  resolved  easier  than  they  are  in  today’s 
system,  the  controller’s  time  orientation  for  detecting  and  resolving  ATC  problems 
will  change. 

Another  perspective  on  the  time  orientation  of  the  controller  under  AERA  was 
prov'ded  by  McKinley  and  Jago  (9).  In  an  early  evaluation  of  the  en  route 
controller’s  skills  requirements,  they  suggested  that  tne  automated  control  capability 
of  AERA  would  only  be  possible  for  iong-term  (greater  than  20  minutes  in  the 
future)  and  some  medium-term  (6-20  minutes)  problem  prediction  and  resolution. 
For  the  short-term  (0-5  minutes)  and  some  medium-term  problem  detection,  reliance 
on  the  information  from  the  situation  display  and  electronic  flight  progress  data 
would  continue  to  be  required. 

Because  AERA  2  and  AERA  3  are  still  under  development,  it  is  difficult  to  assess 
the  cognitive  skills  and  abilities  required  for  the  job  tasks.  However,  it  is  clear  that 
the  implementation  of  AERA  2,  and  particularly  AERA  3,  will  significantly  affect  the 
role  of  the  ATCS  in  controlling  traffic  and  that  research  on  the  impact  must  continue. 


CURRENT  SELECTION  OF  AIR  TRAFFIC  CONTROL  SPECIALISTS 


PERSONNEL  SELECTION  IN  THE  UNITED  STATES 

The  discussion  of  selection  of  controllers  should  be  considered  within  the  context  of 
selection  of  personnel,  in  general,  in  the  United  States.  That  context  is  highly 
regulated,  considerably  politicized,  and  normally  very  public.  Prior  to  the  Civil  Rights 
Act  of  1964,  selection  was  a  backwater  of  personnel  psychology,  including  only  a 
few  individuals  of  talent  and  sensitivity  among  its  ranks.  Civil  rights  activists 
perceived,  mostly  accurately  perhaps,  that  the  selection  procedures  followed  by 
many  companies  bore  little  relationship  to  the  jobs  at  hand  while  they  excluded  high 
percentages  of  minority  applicants.  Much  of  the  early  case  law  on  personnel 
selection  was  built  from  such  cases.  It  could  be  argued  that  many  of  the  resulting 
judicial  opinions  overgeneralized  and,  by  inference,  developed  procedural 
requirements  that  few,  if  any,  of  the  best  selection  systems  could  meet.  Since  the 
1960s,  selection  procedures  have  become  the  subject  of  scrutiny  by  many  different 
groups.  This  has  meant  that  the  procedures  must  be  both  extensively  documented 
and  technically  acceptable.  Because  of  this,  personnel  psychology  left  its  position  in 
the  backwaters  and  has  become  one  of  the  more  exciting  areas  of  scientific 
psychology. 
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The  Uniform  Guidelines  on  Employee  Selection  Procedures,  published  in  1978  (10), 
apply  to  public,  as  well  as  private  employers.  They  include  two  major  requirements: 
the  determination  of  adverse  impact  and  the  demonstration  of  job-relatedness.  If  an 
employer’s  selection  system  shows  adverse  impact  against  one  or  more  selected 
groups,  then  the  job-relatedness  of  this  system  must  be  demonstrated.  The  debates 
surrounding  the  proposed  Civil  Rights  Act  of  1990  have  focused  on  defining  these 
terms  "adverse  impact"  and  "job-relatedness."  For  now,  however,  "adverse  impact" 
can  be  understood  as  the  amount  or  degree  of  differential  effect  that  a  selection 
procedure  has  against  a  protected  group,  that  puts  the  employer  on  the  defensive 
and  initiates  an  investigation  of  the  selection  procedure.  This  is  primarily  a  legal  or 
administrative  matter,  although  psychologists  are  concerned  about 
representativeness  of  samples,  statistical  significance,  and  so  forth. 

Job-relatedness  is  the  concern  of  psychologists.  Over  the  past  25  years  it  has  been 
understood  as  meaning  the  demonstration  of  the  validity  of  the  selection  procedures, 
with  criterion-related  validity  providing  the  most  pe'~Masive  evidence.  Showing  a 
significant  statistical  relationship  between  the  scon  on  a  predictor  test  and  the 
scores  on  some  criterion,  typically  training  grades  or  supervisory  ratings  of  job 
performance,  has  been  convincing  in  many  cases.  Content  validity  can  be 
acceptable  in  certain  circumstances,  such  as  using  a  typing  test  to  select  for  clerical 
occupations.  The  FAA’s  Screen  for  ATCS,  described  below,  is  based  on  both  a 
content  validity  strategy-the  Screen  is  a  miniature  training  and  test  situation  of  how 
a  controller  learns  the  job-and  a  criterion-related  Strategy-Screen  grades  predict 
field  training  performance.  A  third  strategy  is  construct  validity  where  some 
cognitive  ability  (e.g.,  the  construct  of  spatial  orientation  of  objects)  is  posited  as 
necessary  to  the  ATCS  occupation.  Construct  validation  strategies  can  be  risky 
legal  defenses  unless  their  empirical  grounding  is  well-articulated. 

A  major  contribution  to  the  understanding  of  validation  in  personnel  settings  is  the 
concept  of  validity  generalization,  initially  proposed  in  1977  by  Schmidt  and  Hunter 
(11).  These  authors,  and  others  following,  collected  many  criterion-related  studies 
and  used  the  study  results  as  data  in  meta-analyses.  Validity  coefficients  vary 
widely  from  one  study  to  the  next,  a  phenomenon  many  scientists  had  interpreted  as 
supporting  "situational  specificity;"  that  is,  the  use  of  one  predictor  test  cannot  be 
generalized  from  the  situation  where  the  criterion-related  validity  study  was 
performed.  Schmidt  and  Hunter  (7)  showed  that  this  variability  in  validity  coefficients 
might  be  due  not  to  variability  in  situations,  or  employment  settings,  but  to  the 
varying  numbers  of  subjects  in  the  studies,  the  differing  reliabilities  of  criterion 
measures,  and  other  artifacts;  that  is,  criterion-related  validities  do  generalize  to 
other  situations. 

There  have  been  two  major  outcomes  of  the  validity  generalization  movement-and  it 
does  have  some  of  the  characteristics  of  a  movement.  First,  it  has  restored  both 
the  faith  in  and  reliance  on  paper-and-pencil  tests  of  general  mental  abilities  as 
entry-level  predictors  of  success  in  most  jobs.  While  some  argue  that  a  test  of 
general  mental  ability  is  useful  for  any  occupation,  the  historical  distinctions  among 
test  types  (e.g.,  verbal,  quantitative,  spatial)  reflect  small  but  important  differences  in 


9 


predicting  job  success.  Such  generic  paper-and-pencil  tests  are  still  a  major 
component  of  personnel  selection.  Further  distinctions  among  test  types  and  the 
mental  processes  required  to  perform  well  on  them,  such  as  those  being  explored 
by  cognitive  psychologists,  are  usually  of  much  less  interest  because  these 
distinctions  are  not  reflected  in  predictiveness.  However,  those  responsible  for  the 
selection  of  air  traffic  controllers  may  have  more  interest  in  this  cognitive  approach, 
because  of  test  practice  effects  especially  prevalent  in  spatial  ability  tests.  Fairness 
issues  as  well  as  validity  are  paramount  in  this  approach. 

The  other  outcome  of  validity  generalization  has  been  to  focus  more  on  the  criteria 
that  the  tests  have  been  used  to  predict.  Most  of  the  validity  generalization 
research  is  based  on  only  two  classes  of  criteria:  training  success  and  supervisory 
ratings.  By  the  time  the  validity  generalization  research  began  in  the  late  1970s, 
civil  rights  activists  and  some  personnel  psychologists  were  questioning  these  two 
criterion  categories.  While  training  is  important,  so  went  the  argument,  it  is  not  the 
job.  Training  may  require  more  cognitive  processing,  which  paper-and-pencil  tests 
predict  fairly  well,  than  may  the  job.  As  for  supervisory  ratings,  these  are  typically 
generic,  probably  hodge-podge,  and  possibly  biased.  Why,  argued  these  critics,  can 
we  not  validate  our  selection  procedures  against  closer  surrogates  to  actual  job 
performance? 

The  use  of  job  performance  measures  in  criterion-related  validity  research  is  fraught 
with  conceptual  and  logistical  problems  of  high  order.  On  the  conceptual  side, 
should  we  deal  with  "typical"  performance,  that  which  the  employee  does  on  a 
regular  basis  and  that  might  be  measured  by  detailed  supervisor  ratings?  Or  should 
we  try  to  measure  "maximum"  performance,  what  the  employee  can  do,  by 
developing  specialized  situations  in  which  the  desired  performance  can  be 
evaluated?  What  types  of  performance  should  we  measure?  Should  we  measure 
performance  with  a  test  of  knowledge,  or  a  demonstration,  or  a  simulation?  How 
useful  are  archival  measures,  such  as  absenteeism?  Logistically,  most  employees 
are  far  more  scattered  in  time  and  place  than  are  applicants  for  a  job.  Further, 
since  job  incumbents  tend  to  specialize,  how  can  a  single,  albeit  complex, 
performance  measure  be  fair  to  all? 

The  research  on  performance  measurement  is  not  as  advanced  as  that  on  tests  and 
predictors,  but  during  the  last  decade  there  has  been  progress.  The  major  effort 
has  been  a  joint-military  service  effort  sponsored  by  the  U.S.  Department  of 
Defense.  Preliminary  answers  to  the  questions  raised  above  are  (3)  job 
performance  can  be  measured;  (2)  job  performance  measures  can  be  used  in 
criterion-related  validity  research  and  can  be  predicted  by  multiple  abilities  test 
batteries;  (3)  the  most  appropriate  performance  measures  will  differ  from  job  to  job, 
e.g.,  knowledge  tests  would  be  appropriate  for  claims  examiners  but  not  for 
mechanics;  (4)  all  available  performance  measures,  including  archival  measures  can 
be  used  to  describe  a  "performance  space,"  to  provide  a  more  complete  picture  of 
the  occupation  and  its  requirements. 
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An  emphasis  on  performance  assessment,  or  on  criterion-referencing,  has  been 
growing  steadily  during  the  last  two  decades  in  many  areas  of  psychological 
research.  The  most  prominent  example  is  that  of  educational  assessment.  It  no 
longer  is  sufficient  to  have  a  comparison  to  "who,"  the  traditional  basis  of  norms- 
referencing;  it  must  be  a  comparison  to  "what,"  or  criterion-referencing,  and  why  that 
"what"  is  important.  Competency  requirements  for  high  school  graduation  are  an 
example. 

Performance  measures  may  also  be  used  in  ways  not  previously  considered  for 
training  grades  and  ratings,  and  it  is  this  potential  that  is  most  exciting.  For 
example,  let  us  say  a  work  sample  test  were  developed  for  a  subgroup  of  Full 
Performance  Level  (FPL)  air  traffic  controllers.  Say  that  this  measure  could  be 
judged  as  a  representative  sample  of  controller  performance  by  an  appropriate 
sample  of  controllers  and  their  supervisors,  and  that  the  test  scores  could  be  scaled 
into  ranges  such  as  Outstanding,  Exceptional,  Fully  Satisfactory,  Marginal,  and 
Unacceptable.  If  this  performance  measure  has  a  valid  predictor  that  we  would  like 
to  use  in  our  selection  procedures,  we  can  link  the  score  levels  on  the  work  sample 
to  score  levels  on  the  predictor  in  both  a  more  rational  and  more  empirical  manner 
than  is  the  current  practice. 

Thus,  the  state  of  the  art  in  selection  research  is:  (2)  It  is  strictly  regulated,  very 
public,  heavily  documented,  and  of  mostly  high  quality;  (2)  Using  paper-and-pencil 
tests  of  general  abilities  has  much  technical  justification;  and  (3)  Performance 
measures  are  becoming  much  more  necessary  in  validation  of  selection  procedures. 
It  is  likely  that  computers  will  play  an  increasing  role  in  selection  research  and  in 
selection,  as  they  have  in  other  areas  of  human  endeavor.  However,  the  question 
for  the  technology  will  be  the  technical  adequacy  of  the  selection  research  program 
in  which  it  is  embedded.  Research  in  selection  of  air  traffic  controllers  of  the  future 
will  take  place  in  this  environment  and  must  meet  the  existing  standards  and 
regulations  for  employee  selection. 


SELECTION  OF  AIR  TRAFFIC  CONTROLLERS 


History  of  the  development  of  ATCS  selection  procedures. 

The  FAA  has  chosen  to  utilize  a  selection  system  prior  to  investing  in  training 
individuals  as  ATCSs.  The  reason  for  the  decision  was  a  high  attrition  rate  from  the 
occupation  when  no  formal  selection  system  was  utilized  prior  to  entry  into  training. 
The  attrition  was  generally  due  to  training  failure  and  occurred,  on  the  average,  two 
to  three  years  into  a  person’s  career  in  air  traffic  (12).  Such  late  attrition  resulted  in 
considerable  costs  to  both  the  FAA  and  the  ATCS,  who  found  little  opportunity  to 
apply  the  ATC  skills  in  other  occupations.  Toward  the  goal  of  reducing  attrition  as 
well  as  identifying  the  individual  characteristics  required  for  success  in  the 
occupation,  the  FAA  developed  an  active  selection  research  program. 
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As  described  by  Brokaw  (13),  research  on  ATCS  selection  dates  back  to  the  1950s 
and  his  contract  with  the  Civil  Aeronautics  Administration  (CAA--precursor  to  the 
FAA)  for  the  development  of  a  paper-and-pencil  aptitude  test  battery  that  could  be 
used  to  select  ATCS  trainees.  The  results  of  the  study  indicated  that  some  aptitude 
tests  reflecting  content  areas  identified  from  a  job  analysis  could  potentially 
contribute  to  the  selection  process.  (This  study  provided  the  format  for  an  Air 
Traffic  Problems  test).  In  a  joint  effort  with  the  CAA,  the  United  States  Air  Force 
Personnel  Laboratory  administered  37  aptitude  tests  to  ATCS  trainees  in  1956.  The 
findings  indicated  that  a  composite  aptitude  test  score-created  by  adding  scores  on 
tests  of  arithmetic  reasoning,  symbolic  reasoning,  code  translation,  and  the  ATP 
test-effectively  predicted  instructor’s  ratings  of  training  performance  and  supervisors’ 
ratings  of  job  performance  approximately  one  year  after  training. 

Research  continued  in  1960  with  the  establishment  of  the  FAA  Civil  Aeromedical 
Research  Institute  (presently  the  Civil  Aeromedical  Institute-CAMI),  as  described  by 
Collins,  Boone,  and  VanDeventer  (14).  CAMI  research  on  Civil  Service  Commission 
(CSC)  tests  of  similar  content  to  the  tests  originally  studied  by  Brokaw  led  the  CSC 
for  the  first  time  to  establish  tests  for  selecting  ATCS  trainees.  Test  requirements 
for  selection  were  implemented  in  1962.  The  initial  selection  test  battery  consisted 
of  the  following:  Arithmetic  Reasoning,  Spatial  Relations,  Following  Oral  Directions, 
Abstract  Reasoning,  and  Air  Traffic  Problems.  During  the  period  between  1962  and 
1972,  in  addition  to  continuing  validation  research,  aviation  psychologists  studied  a 
number  of  factors  relevant  to  ATCS  selection:  attrition  from  the  profession,  age, 
prior  experience,  education,  sex,  and  military  ATC  training. 

Implementation  of  the  CSC  test  battery  alone  failed  to  sufficiently  decrease  the 
attrition  rate.  Prior  to  1976,  attrition  rates  from  field  training  were  38%.  A  number 
of  studies  were  completed  in  the  early  1970s,  as  well  as  Congressional  hearings  of 
the  Committee  on  Government  Operations,  which  resulted  in  several 
recommendations.  As  a  result  of  the  reports,  the  FAA  developed  a  standardized, 
centralized,  validated  program  designed  to  identify,  as  early  as  possible,  and  remove 
from  training  those  candidates  demonstrating  insufficient  aptitude  to  become  ATCSs. 
The  program  was  designed  to  decrease  the  costs  of  attrition  and  improve  the 
selection  of  ATCSs.  The  FAA’s  aviation  psychologists  collaborated  to  provide  a 
scientific  framework  required  for  the  design,  implementation,  and  evaluation  of  the 
program. 

The  result  was  the  implementation  of  a  second-stage  selection  procedure  designed 
to  follow  the  CSC  test  battery.  This  second-stage  hurdle  consisted  of  two  option- 
specific  (en  route  and  terminal)  pass/fail  programs  utilizing  courses  similar  to  those 
used  in  the  nonradar  separation  portions  of  the  FAA  Academy  ATC  training 
curriculum  existing  at  that  time.  Grades  in  these  courses  had  been  found  to  be 
predictive  of  future  field  attrition  (12).  The  programs  were  implemented  in  1976. 

During  the  latter  half  of  the  1970s  a  new  initial  selection  test  was  developed. 
Research  focused  on  studies  relevant  to  the  Uniform  Guidelines  on  Employee 
Selection  Procedures,  and  on  developing  field  training  performance  ratings, 
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developing  a  longitudinal  database  for  continuing  validation  research,  and  refining 
optimal  combinations  of  old  and  new  aptitude  screening  measures  (14). 


The  current  ATCS  selection  system. 

The  current  ATCS  selection  system  consists  of  two  hurdles  or  stages  of 
assessment.  The  following  sections  present  these  procedures  in  more  detail  and 
give  statistics  on  each  method's  ability  to  predict  other  measures  of  ATCS 
performance. 


The  PPM  Selection  Battery.  The  first  stage  of  the  selection  process  is  the  Office  of 
Personnel  Management  Air  Traffic  Control  selection  test  battery.  This  stage  is  a 
paper-and-pencil  format  aptitude  test  battery  consisting  of  three  tests:  the  Multiplex 
Controller  Aptitude  Test  (MCAT),  Abstract  Reasoning  (included  from  the  prior 
selection  battery),  and  the  Occupational  Knowledge  Test  (OKT).  It  provides  the 
usual  benefits  of  paper-and-pencil  tests,  allowing  initial  assessment  of  large  numbers 
of  potential  candidates  at  relatively  low  cost. 

The  MCAT  was  developed  to  establish  a  paper-and-pencil  test  with  higher  predictive 
validity  than  tests  included  in  the  CSC  selection  battery.  The  MCAT  was  designed 
to  approximate  simulated  air  traffic  activities.  Simulated  airspaces  depict  several 
aircraft  traversing  the  space.  An  altitude,  speed  and  route  chart  is  also  presented. 
Candidates  are  asked  to  identify  situations  that  may  result  in  conflicts  between 
aircraft  based  upon  a  limited  set  of  separation  criteria.  Other  items  require 
computing  time-distance  functions,  interpreting  information,  and  recognizing  spatial 
relations.  The  test  is  in  a  timed,  multiple-choice  format.  The  Abstract  Reasoning 
measures  two  principles  of  logical  development  exhibited  by  sequences  of  figures 
and  letters  (14).  The  Occupational  Knowledge  Test  (OKT)  provides  a  more 
objective  and  reliable  measure  of  a  candidate's  ATCS  job  knowledge  than  that  which 
was  provided  by  ratings  based  upon  job  history.  Items  on  the  OKT  cover  seven 
knowledge  areas  related  to  air  traffic  control. 

This  battery,  as  is  true  of  many  others,  including  spatial  ability  tests,  is  highly 
subject  to  practice  effects.  Generally,  a  second,  higher  score  has  lower  validity  than 
the  first  score.  To  compensate  for  practice  effects,  applicants  earning  a  score  of  70 
or  above  (out  of  a  possible  100)  presently  cannot  retake  the  OPM  Selection  Battery 
for  18  months. 


The  FAA  Academy  Screen.  The  second  stage  of  the  selection  process  is  the  FAA 
Academy  Screen,  a  nine-week  assessment  program  equivalent  to  a  miniaturized 
training-testing  environment.  Candidates  are  taught  rules  for  aircraft  separation  and 
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are  assessed  in  laboratory  simulation  problems  on  their  ability  to  learn  and  apply  the 
rules.  This  pass/fail  stage  is  very  resource-intensive,  but  selects  candidates  likely  to 
succeed  to  full  performance  level  from  among  those  who  have  passed  the  initial 
OPM  battery. 

The  goal  of  the  second  stage  program,  which  was  implemented  in  1976,  was  to 
reduce  field  attrition  through  early  identification  of  candidates  with  little  chance  of 
succeeding  in  field  training.  Because  the  Screen  is  a  miniaturized  training-testing 
assessment  program,  its  purpose  is  to  teach  a  candidate,  with  no  knowledge  of  air 
traffic  control,  enough  about  the  job  to  assess  the  potential  to  succeed  to  the  full 
performance  level.  This  is  accomplished  by  training  the  candidate  in  aspects  of  the 
job  and  assessing  performance  on  a  sample  task  (i.e.,  separating  aircraft  in 
laboratory  nonradar  simulation  problems).  The  complexities  of  the  laboratory 
problems  are  structured  to  escalate  quickly,  generally  beyond  most  candidates’ 
ability  to  master  the  complexity  levels  being  tested.  This  allows  for  a  certain  level  of 
discrimination  of  the  better  candidates  from  the  candidates  with  less  ability. 

The  program  includes  four  weeks  of  didactic,  classroom  training  on  nonradar  ATC 
rules  and  principles.  Content  includes  rules  of  separation  (vertical,  longitudinal,  and 
lateral),  cooperative  agreements,  protection  of  special-use  airspace,  phraseology, 
and  flight  progress  strip-marking.  Seme  practice  in  applying  the  rules  is  provided  in 
the  classroom.  For  the  final  five  weeks,  training  focuses  on  application  of  the  rules 
in  a  nonradar  laboratory.  Each  student  performs  nonradar  controller  job  tasks  in 
one  or  two  thirty-minute  scripted  scenarios  each  day.  During  each  scenario  an 
individual  instructor  is  assigned  to  each  student.  Following  the  scenario,  the 
instructor  reviews  the  student’s  performance  with  the  student,  providing  feedback  on 
errors,  correct  application  of  rules,  and  other  required  skills. 

Thirteen  performance  measures  are  made  of  each  candidate  during  the  nine-week 
period.  Five  of  these  assessments  are  multiple-choice  tests  on  the  academic 
portion  of  the  Screen.  These  assess  the  candidate’s  ability  to  learn  and  retain  the 
basic  knowledge  required  for  the  job.  A  sixth  measure  is  a  map  test  which 
assesses  the  students’  ability  to  learn  a  map  of  the  relatively  simple  synthetic 
airspace.  These  measures  account  for  20%  of  the  final  grade.  Six  of  the 
performance  measures  are  formal  evaluations  of  the  student’s  performance  during 
six  30-minute  standardized  scenarios.  The  score  on  these  laboratory  simulation 
problems  consists  of  a  technical  assessment  (number  of  errors)  and  an  instructor 
assessment  (subjective  evaluation  of  student  performance).  Only  five  of  the  six 
laboratory  problems  are  included  in  calculating  the  final  score.  These  five  scores, 
however,  comprise  60%  of  the  final  grade.  Finally,  the  students  are  given  a  timed, 
multiple-choice  test,  the  controller  skills  test  (CST),  which  is  a  paper-and-pencil 
assessment  of  the  students’  ability  to  apply  ATC  rules  and  procedures.  The  CST 
comprises  20%  of  the  final  composite  grade.  A  final  grade  of  at  least  70  (out  of  a 
possible  score  of  100)  is  required  to  pass  the  Screen.  Failure  to  achieve  a 
composite  score  of  70  results  in  removal  from  the  ATCS  job. 
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The  ATCS  selection  procedures  and  field  training  status  and  attrition  rates. 


As  noted,  the  Academy  screening  programs  were  implemented  in  1976  to  provide 
early  identification  and  removal  of  candidates  with  little  chance  of  success  in  the 
occupation  and  thus  reduce  field  attrition  rates.  Prior  to  1976,  the  attrition  rate  in 
field  training  was  38%.  After  implementation  of  the  screening  programs,  attrition 
rates  at  the  Academy  averaged  30%  and  field  training  attrition  declined  to  8%  (15). 
Following  the  1981  controllers’  strike  and  subsequent  increase  in  the  population  of 
Academy  entrants  with  no  prior  ATC  experience,  the  attrition  rate  at  the  Academy 
increased  to  40%,  while  the  field  attrition  rate  increased  to  11%.  Thus,  the 
Academy  screening  programs  were  effective  in  decreasing  attrition  from  field 
training. 

To  determine  the  extent  to  which  the  original  OPM  and  Screen  measures  predicted 
subsequent  measures  of  performance  in  field  training  in  the  en  route  (ARTCC) 
environment,  Manning,  et  al,  (16)  compared  correlations  between  the  OPM  ATCS 
selection  test  scores  and  scores  from  the  FAA  Academy  Screen  with  measures  of 
field  training  performance  for  competitive  Academy  students  who  entered  between 
September  1981  and  September  1985.  Of  the  OPM  measures,  the  MCAT  was 
found  to  be  a  better  predictor  of  field  training  status  in  the  en  route  option  than 
other  OPM  tests.  Academy  laboratory  performance  scores  were  found  to  be  more 
predictive  of  field  training  performance  than  the  academic  test  scores.  Finally,  for 
the  en  route  option,  Academy  performance  measures,  particularly  laboratory  scores, 
were  better  predictors  of  supervisor  and  on-the-job  training  (OJT)  instructor  ratings 
and  training  status  than  were  OPM  scores. 


Current  correlations  between  performance  on  the  OPM  battery.  Academy  Screen, 
and  field  training  status. 

In  October  1985,  a  change  was  made  to  the  Academy  Screen  which  resulted  in 
reweighting  the  performance  measures.  Because  the  average  time  for  an  en  route 
ATCS  to  attain  full  performance  level  (FPL)  status  is  just  under  3  years,  we  are  just 
now  able  to  evaluate  the  impact  of  this  change  on  predictive  validity.  The  analyses 
reported  here  are  an  initial  evaluation  of  the  change.  Because  this  paper  has 
primarily  focused  on  how  increasing  automation  affects  the  en  route  controller,  the 
analyses  address  only  the  en  route  option.  In  addition,  a  modification  to  the  OPM 
test  battery  administration  procedures  was  also  implemented  in  October  1985. 
Because  the  affect  of  the  change  has  not  yet  been  evaluated,  this  paper  presents 
only  data  from  ATCSs  who  took  the  version  of  the  OPM  test  prior  to  the 
modification. 


To  assess  the  validity  of  the  new  Academy  Screen,  analyses  were  conducted  to 
examine  the  distributions  of  OPM  and  Academy  scores  for  first-time  competitive 
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students  who  entered  the  Academy  between  October  1985  and  September  1986, 
(i.e.,  during  FY-1986).  As  noted,  the  sample  included  only  students  with  OPM 
ratings  from  the  former  administration  procedures  and  only  those  students  assigned 
to  en  route  facilities.  Finally,  ATCS  developmental  were  excluded  from  the  sample 
if  they  withdrew  from  the  career  field  for  reasons  other  than  failing  training  (e.g.,  to 
pursue  another  career). 


Appendices  B-E  show  statistics  related  to  the  distributions  of  OPM  test  scores  and 
Academy  performance  measures.  Means  and  standard  deviations  of  OPM  test 
scores  for  all  applicants  who  took  the  OPM  test  between  April  and  October  1985 
are  shown  in  Appendix  B.  Means  and  standard  deviations  of  OPM  test  scores  for 
those  who  entered  the  Academy  between  October  1985  and  September  1986  and 
took  the  prior  administration  of  the  OPM  test  are  shown  in  Appendix  C.  Means  and 
standard  deviations  for  Academy  performance  measures  are  shown  in  Appendix  D. 
Appendix  E  shows  correlations  between  OPM  test  scores  and  Academy  performance 
measures. 


Tables  1  and  2  show  correlations  of  OPM  and  Academy  performance  measures  with 
a  variable  measuring  status  in  field  training,  both  unadjusted  and  adjusted  for 
restriction  in  the  range  of  predictor  scores,  respectively.  The  variable  "field  training 
status"  has  the  following  categories:  1)  reached  FPL,  2)  still  in  training,  3)  switched 
options,  and  4)  failed.  The  tables  compare  the  correlations  reported  in  the  Manning, 
et  al,  (16)  study  with  those  computed  for  the  FY-1986  sample  of  Academy  Screen 
graduates  assigned  to  the  en  route  option.  It  can  be  seen  that  the  correlations  are 
similar.  Examination  of  Table  2  shows  that  the  correlation  between  the  OPM  rating 
and  field  training  status,  corrected  for  restriction  in  the  range  of  OPM  rating,  is 
higher  than  was  found  for  the  1981-1985  sample,  although  the  correlations  of 
individual  OPM  component  scores  with  the  criterion  were  similar. 
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TABLE  1 

CORRELATIONS  BETWEEN  OPM  AND  ACADEMY  SCREEN  PERFORMANCE 

MEASURES 

AND  MEASURES  OF  FIELD  TRAINING  STATUS 
For  1981-1985  En  Route  and  1986  Screen  Entrants 
(Uncorrected  for  restriction  in  range) 


1981-1985 

1986 

En  Route 

Screen 

OPM  TEST  MEASURES 

Multiplex  Controller 

(n=2992) 

(n=402) 

Aptitude  Test  (MCAT) 

-.12* 

-.09 

Abstract  Reasoning 
Occupational  Knowledge 

.03 

-.03 

Test  (OKT) 

.00 

-.05 

Transmuted  Composite 
(TMC) 

-.10* 

-.08 

OPM  Rating 

-.05* 

-.09 

ACADEMY  SCREEN  MEASURES 

Block  Test  Average 

-.05* 

-.06 

Controller  Course  Test 
Ave.  5  of  6  Laboratory 

-.04 

-.07 

Problems 

Ave.  Instructor 

-.24* 

-.21* 

Assessment 

Ave.  Technical 

-.25* 

-.22* 

Assessment 

Controller  Skills 

-.22* 

-.21* 

Test 

-.08* 

-.16* 

Comprehensive  Score 

-.24* 

-.24* 

Correlations  were  significantly  different  from  0  at  p<.01. 


TABLE  2 

CORRELATIONS  BETWEEN  OPM  AND  SCREEN  PERFORMANCE  MEASURES 
AND  MEASURES  OF  FIELD  TRAINING  STATUS 
For  1981-1985  En  Route  and  1986  Screen  Graduates 
(Adjusted  for  restriction  in  the  range  of  predictor  scores) 


1981-1985  1986 

En  Route  Screen 

(n=2992)  (n=402) 


OPM  TEST  MEASURES 


Multiplex  Controller 
Aptitude  Test 

-.28 

-.24 

Abstract  Reasoning 

.05 

-.04 

Occupational  Knowledge 
Test 

.00 

-.04 

Transmuted  Composite 

-.25 

-.24 

OPM  Rating 

-.13 

-.30 

ACADEMY  SCREEN  MEASURES 

Block  Test  Average 

-.09 

-.10 

Controller  Course 

Test 

-.05 

-.09 

Ave.  5  of  6  Laboratory 
Problems 

-.42 

-.36 

Ave.  Instructor 
Assessment 

-.46 

-.37 

Ave.  Technical 
Assessment 

-.33 

-.30 

Controller  Skills  Test 

-.14 

-.26 

Comprehensive  Score 

-.46 

-.44 

Also,  the  correlations  of  the  Laboratory  Average  (average  score  on  best  5  of  6 
laboratory  problems)  and  Laboratory  Instructor  Assessment  (average  score  on 
instructor  subjective  rating  in  laboratory  problems)  with  en  route  field  training  status 
decreased  somewhat,  while  the  correlation  between  the  CST  (Controller  Skills  Test) 
and  en  route  field  training  status  increased  somewhat.  For  the  remaining  measures, 
the  similarity  in  correlation  coefficients  for  the  two  samples  suggests  that  the  validity 
of  the  program  in  predicting  status  in  field  training  has  remained  stable,  in  spite  of 
the  changes  made  to  the  Academy  Screen. 
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Use  of  Nonradar  ATC  Tasks  to  Predict  a  Radar-based  Job 


The  Screen  program  is  based  upon  nonradar  air  traffic  separation  rules.  At  the  time 
it  was  developed,  nonradar  procedures  were  used  more  frequently.  Today,  radar 
separation  procedures  are  utilized  almost  exclusively  in  the  majority  of  the  ARTCC’s. 
Thus,  the  question  arises  about  the  extent  to  which  a  program  that  is  based  on 
nonradar  procedures  may  effectively  predict  success  in  a  radar  environment. 

A  comparison  of  the  ATC  tasks  trained  in  the  Academy  Screen  to  the  tasks  required 
of  an  en  route  controller  suggests  that  many  of  the  behaviors  measured  in  the 
current  Screen  are  similar  to  those  required  in  the  radar  environment.  Of  the  six 
activities  of  an  en  route  controller  identified  in  the  CTA  job  task  analysis  (1.  perform 
situation  monitoring,  2.  resolve  aircraft  conflicts,  3.  manage  air  traffic  sequences,  4. 
route  or  plan  flights,  5.  assess  weather  impact,  and  6.  manage  sector/position 
resources),  the  Screen  assesses  tasks  associated  with  five  of  the  six  activities. 
Only  weather  assessment  (activity  5)  is  not  taught  or  tested  in  the  Screen  program. 

With  the  exception  of  the  tasks  that  require  checking  the  plan  view  display  (PVD)  or 
utilizing  radar  separation  procedures,  most  of  the  tasks  in  the  composition  graph  are 
similar  to  the  nonradar  training  taught  and  evaluated  in  the  Screen.  For  example, 
the  first  sub-activity  under  "perform  situation  monitoring"  is  checking  and  evaluating 
the  separation  of  aircraft.  The  sequence  of  tasks  for  this  sub-activity  proceeds  as 
follows:  i.  Review  of  flight  progress  for  present  and/or  future  aircraft  separation;  ii. 
Review  of  PVD  (or  flight  progress  strips  in  nonradar)  for  potential  violation  of 
airspace  separation  standards;  iii.  Review  of  location  of  flight  progress  strip  in  bay,  if 
unsatisfactory,  resequence  manually;  iv.  Mental  assessment  of  potential  conflict- 
project  mentally  an  aircraft’s  future  position/altitude/path  and  evaluate  mental  flight 
plan  projection  for  appropriateness;  v.  Range/bearing  data  may  assist  assessment  of 
possible  conflict-request  range/bearing/time  (by  contacting  the  pilot  in  nonradar).  Of 
18  tasks  described  for  the  sub-activity,  12  tasks  or  nonradar  correlates  are  assessed 
in  the  Screen.  This  is  an  example  of  the  extent  to  which  the  tasks  taught  and 
assessed  in  the  Screen  can  be  related  to  the  tasks  required  of  an  en  route 
controller  even  though  radar  equipment  has  added  an  automated  dimension  and 
new,  specific  air  traffic  separation  procedures. 

Knowing  that  performance  on  nonradar  tasks  has  some  utility  in  predicting  future 
success  in  the  more  highly-automated  radar-based  occupation  may  help  develop 
selection  procedures  for  the  automated  systems  of  the  future.  Perhaps  selection 
procedures  designed  for  the  current  occupation  will  predict  performance  in  the 
evolving  occupation. 

Because  of  the  ATCS  strike  in  1981,  development  of  a  radar-based  selection 
program  was  delayed.  Recently,  however,  interest  and  enthusiasm  for  addressing 
questions  of  new  and  improved  selection  procedures  for  ATCSs  have  been  revived. 
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COMPUTERIZED  SELECTION  SYSTEMS 


Since  the  development  of  the  FAA's  selection  procedures,  the  use  of  computers  in 
assessment  has  become  more  feasible  and  may  have  some  utility  for  future 
selection  of  ATCSs.  Much  interest  and  effort  during  the  last  two  decades  have 
been  devoted  to  creating  computer-based  assessment  and  selection  systems.  A 
number  of  computer-based  assessment  systems  are  available  commercially.  Some 
are  directed  towards  clerical  occupations  while  others  are  marketed  for  both 
vocational  guidance  and  selection,  using  computer-adaptive  multiple  cognitive 
abilities  batteries. 

Further  selection  applications  have  been  few.  However,  several  problems 
complicate  the  application  of  computerized  testing  to  the  selection  of  employees. 
First,  typical  test  construction  problems  of  standardization  also  apply  to  computerized 
tests,  as  well  as  the  computer-specific  problems  of  calibration  and  user-friendliness. 
Second,  when  an  organization  has  computers,  it  is  not  hard  to  justify  the  use  of 
some  of  them  for  testing.  When  the  selection  function  requires  its  own  set  of 
computers,  it  is  difficult  for  the  organization  to  justify  the  use  of  all  of  them  in 
testing.  Thus,  large-scale  computerized  selection  programs,  such  as  might  be 
expected  in  the  military  services,  have  been  difficult  to  implement.  Third,  the 
selection  research  base  for  abilities  measurable  only  by  computer,  such  as  dynamic 
spatial  ability  and  processing  speed,  remains  comparatively  slim.  The  three  military 
research  laboratories  in  the  United  States  have  each  developed  computerized 
assessment  systems,  which  measure  abilities  different  from  those  assessed  by 
paper-and-pencil  selection  tests.  As  this  research  base  develops,  computers  are 
likely  to  play  a  larger  role  in  selection. 


FUTURE  SELECTION  SYSTEMS 

This  paper  has  presented  an  overview  of  the  NAS  Plan  for  automation  of  the  air 
traffic  controllers’  job  over  the  next  two  decades.  To  the  extent  that  changes  in  job 
tasks  have  been  projected  so  far,  we  have  attempted  to  map  them  for  the  Initial 
Sector  Suite  System  and  Automated  En  Route  Air  Traffic  Control.  At  this  time,  it 
appears  that  the  job  of  the  ISSS  controller  will  require  many  of  the  same  skills  and 
cognitive  attributes  required  today.  The  implementation  of  AERA,  however,  should 
begin  to  substantially  change  the  involvement  of  the  controller  from  a  tactical  mode 
of  operation  to  a  more  strategic  mode. 

Clearly,  much  of  the  controller’s  activity  when  operating  AERA  2  will  involve 
examining  data,  testing  resolutions,  and  communicating  with  the  computer,  aircraft, 
or  other  controllers.  While  a  controller  will  still  need  to  be  able  to  visualize 
interrelationships  between  aircraft,  the  need  to  make  rapid  decisions  using  the 
information  will  be  reduced.  The  description  of  controller  activities  provided  by 
MITRE  suggests  that  it  might  be  appropriate  to  select  controllers  with  different  types 
of  skills,  emphasizing  strategic  instead  of  tactical  planning.  This  would  suggest  that 
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a  different  battery  might  be  appropriate  for  selecting  AERA  2  ATCSs. 


The  problem  with  the  description  of  the  future  controller  being  exclusively  one  who 
executes  the  computer’s  suggestions  is  that  different  skills  may  be  required  if  control 
actions  are  required  in  a  short-term  time  frame,  perhaps  a  shorter  time  frame  than 
the  computer  can  resolve,  or  during  emergencies.  For  example,  if  events  (e.g.,  VFR 
popups)  occur  that  are  not  identified  by  the  computer,  then  the  controller  may  be 
required  to  respond  quickly  to  avoid  loss  of  separation  between  aircraft.  These 
actions  are  complicated  by  the  proposed  "direct  routing"  of  aircraft  from  one 
destination  to  another,  without  using  designated  airways.  If  emergencies  requiring 
controllers  to  engage  in  tactical  control  activities  occur  only  rarely,  and  if  tactical 
ATC  is  be  complicated  by  direct  routing,  then  perhaps  the  future  controller  will 
require  skills  at  separating  traffic  that  exceed  those  required  today.  As  long  as  the 
controller  retains  responsibility  for  the  movement  of  all  aircraft  in  the  assigned 
airspace  (even  though  the  computer  ordinarily  provides  resolutions  to  potential 
conflicts),  then  he  or  she  will  have  to  possess  and  be  able  to  maintain  the  KSAs  to 
"manually"  separate  ihose  aircraft,  in  case  of  emergency.  Therefore,  in  this  case, 
the  selection  of  ATCSs  must  be  based  on  the  requirement  to  perform  under  the 
worst-case  scenario.  If,  over  time,  the  AERA  software  is  accepted  as  accurate,  and 
is  found  to  be  able  to  handle  all  emergency  situations,  then  the  responsibility  for  the 
aircraft  may  be  removed  from  the  ATCS  and  assigned  to  the  computer,  and  the 
controllers’  job  requirements  may  be  changed,  resulting  in  modifications  to  selection 
procedures. 

As  plans  for  AERA  3  become  mjie  fully  developed,  the  tradeoff  of  separation 
responsibilities  between  the  ATCS  and  the  computer  will  become  more  pronounced. 
Preliminary  projections  am  that  the  AERA  3  software  will  undertake  all  control 
activities,  perhaps  leaving  the  ATCb  1c  issue  clearances  to  aircraft  that  do  not  have 
data  link  capabilities,  or  perform  traffic  management  activities.  The  controller  may 
also  be  responsible  for  ensuring  that  the  computer  is  operating  properly.  In  all 
cases,  even  through  AERA  3,  the  computer  is  dependent  on  timely,  accurate  input 
of  information  from  human  operators  and  is,  thus,  vulnerable  to  the  reliability  and 
accuracy  of  the  human.  Again,  if  the  controller  has  responsibility  for  taking  over  the 
control  of  air  traffic  when  (if)  the  computer  fails,  then  the  selection  procedures  must 
identify  those  who  have  the  ability  to  control  traffic,  even  if  they  only  do  so  on  rare 
occasions.  If,  on  the  other  hand,  the  full  responsibility  for  aircraft  separation  lies 
with  the  computer  (perhaps  yet  another  level  of  software),  then  selection  procedures 
can  be  targeted  to  the  KSAs  required  to  perform  the  duties  most  frequently 
performed  by  the  ATCS  using  AERA  3. 

Development  of  any  selection  procedure  depends  on  an  accurate  description  of  the 
tasks  to  be  performed  and  the  knowledge,  skills,  and  abilities  (KSAs)  required  to 
perform  those  tasks.  The  problem  with  applying  this  approach  to  the  development 
of  selection  procedures  for  the  "automated  controller"  is  obtaining  a  sufficiently 
detailed  job/task  analysis  prior  to  implementation  of  the  new  system,  because  no 
controllers  will  have  used  the  system  extensively  enough  to  qualify  as  subject  matter 
experts  (SME’s).  A  fallback  position  would  be  to  work  with  controller  members  of 
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the  Air  Traffic  AERA  Concepts  Team  (ATACT),  who  are  as  knowledgeable  as  any 
controllers  about  the  way  the  new  system  will  work.  These  "near"  SMEs  could 
speculate  about  the  requirements  to  perform  the  tasks.  Preliminary  work  can  be 
done  early  in  the  development  cycle  to  identify  or  develop  tests  to  measure  the 
KSAs,  as  they  are  currently  identified.  Review  cycles  will  have  to  be  built  into  the 
process  to  allow  periodic  review  of  the  KSAs  with  SMEs  as  more  controllers  are 
brought  into  the  loop  to  test  the  AERA  software  and  procedures,  and  as  the 
software  and  procedures  become  increasingly  well-defined.  Using  such  an  iterative 
process,  a  tentative  set  of  selection  procedures  would  be  available  upon 
implementation  of  the  new  system. 

When  contemplating  the  development  of  selection  procedures  for  employees  who 
will  be  ATCSs  under  the  AERA  levels  of  automation,  it  is  clear  that  work  must  begin 
now  for  the  procedures  to  be  ready  when  AERA  2  software  is  installed.  At  the 
same  time,  the  division  of  duties  between  the  software  and  the  controller  have  not 
yet  been  finalized,  so  the  final  choice  of  a  selection  procedure  is  premature. 

Because  of  the  criticality  of  the  ATCS  occupation,  it  may  not  be  possible  to  select 
everyone  who  applies  for  the  job  and  passes  the  new  selection  procedure  for  a 
period  of  time  until  the  validity  of  the  new  selection  battery  can  be  established. 
When  converting  to  the  AERA  system,  it  may  be  necessary  to  also  continue  using 
the  old  selection  procedures  for  awhile,  until  the  validity  of  the  new  procedures  can 
be  evaluated.  The  correlations  between  measures  of  performance  in  the  nonradar 
Screen  and  success  in  radar-based  field  training  suggest  that  using  a  selection 
procedure  current,  for  example,  for  the  radar  job  to  select  controllers  for  the  new 
ATC  job  under  ISSS  may  be  an  acceptable  interim  solution  while  validation  of  the 
new  procedures  occurs. 

Furthermore,  several  years  may  be  required  to  establish  the  validity  of  new  selection 
procedures.  Most  of  that  time  is  needed  for  those  selected  to  complete  field  training 
so  that  on-the-job  performance  can  be  measured,  and  the  relationships  between 
selection  predictors  and  job  performance  criteria  can  be  assessed.  Measures  of 
performance  in  training  can  be  used  as  interim  criteria,  but,  as  previously  mentioned, 
are  less  desirable  than  measures  of  job  performance. 


DISCUSSION 

The  research  from  the  past  30  years  has  produced  some  findings  that  can 
contribute  to  the  development  of  selection  procedures  for  automated  systems.  First, 
paper-and-pencil  aptitude  tests  alone  have  been  found  insufficient  to  predict 
performance  on  the  complex  procedural  tasks  involved  in  ATC  in  its  current 
environment.  Second,  the  addition  of  a  second-stage  miniaturized  training-testing 
program  has  improved  the  utility  and  validity  of  the  ATCS  selection  process  and 
reduced  field  attrition.  This  selection  program,  based  upon  nonradar  separation,  has 
been  useful  for  predicting  success  in  the  radar  environment. 
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As  with  selection  research  in  other  occupations,  the  research  on  air  traffic  control 
specialists  suffers  from  lack  of  availability  of  criterion  measures  of  ATC’s  job 
performance.  Because  the  Screen  measures  task  samples  via  the  laboratory 
problems,  the  experience  gained  in  developing  measures  of  training  performance 
for  both  the  Screen  and  the  Academy  basic  radar  training  courses  could  contribute 
to  future  efforts  to  develop  criterion  job  performance  measures  for  the  automated 
ATC  environment. 

The  current  selection  system  has  been  found  to  have  utility  in  decreasing  the  field 
attrition  rate;  however,  it  may  not  be  the  best  selection  system  for  radar  ATCSs  or 
the  evolving  automated  occupation,  and  it  is  certainly  very  costly  in  terms  of 
monetary  and  human  resources.  Therefore,  the  FAA  is  beginning  to  explore 
alternatives  to  the  current  methods.  The  goals  are  to  have  a  shorter  and  less  costly 
selection  process,  with  predictive  validity  similar  to  the  present  system,  in  place 
within  the  next  five  years. 

Clearly,  the  development  of  selection  procedures  for  automated  systems  will  be  a 
difficult  process,  complicated  by  a  lack  of  information  about  the  job,  varying 
expectations  about  eventual  job  task  requirements,  extensive  time  frames  for 
validation,  and  a  need  to  develop  appropriate  criterion  measures  of  job  performance. 
The  time  to  start  addressing  these  issue  is  now,  so  that  development  of  selection 
procedures  for  the  changing  occupation  can  parallel  the  evolution  of  the  job. 
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APPENDIX  A 


Cognitive/Sensory  Attribute  Definitions  from  Computer  Technology  Associates’  Job 
Task  Analysis  (4). 

CODING  Transformation  or  translation  of 

information  for  entry  into  the  system; 

Converting  textual  information  to 
graphics  or  symbols. 

Example:  Entering  a  PIREP  (pilot  report);  Composing  a 
flight  plan  amendment. 

DECODING  Transformation  or  translation  of 

information  received. 

Example:  Recognizing  a  symbol  for  a 

handoff;  Reading  a  Flight  Data  Entry. 

DEDUCTIVE  Ability  to  reach  a  conclusion  that 

REASONING  follows  logically  from  the  known  facts 

or  data;  Selection  from  among 
alternative  answers  or  methods. 

Example:  Concluding  that  two  aircraft 
are  on  intersecting  paths 

FILTERING  Selection  of  inputs  on  which  to  focus 

attention  in  the  presence  of  distracting 
stimuli  or  high  workload;  Selective 
attention;  Overload  accommodation. 

Example:  Identifying  communication  transmissions 

for  attention  during  a  period  of  heavy  radio  traffic. 

IMAGE/  Perception  of  spatial  patterns  and 

PATTERN  relations  among  static  or  dynamic 

RECOGNITION  visual  inputs.  May  involve  orienting 

oneself  to  the  picture  or  configuration. 

Example:  Forming  a  picture  of  the  traffic  situation 

by  reviewing  Flight  Data  Entries  on  the  Flight 
Data  Display. 


INDUCTIVE  Generation  of  an  explanation  for  a  set 

REASONING  of  specific  data  or  instances,  giving 


A 1 


INDUCTIVE 
REASONING  (cont.) 


Example: 


LONG-TERM 

MEMORY 


Example: 


MATHEMATICAL/ 

PROBABILISTIC 

REASONING 


Example: 

MOVEMENT 

DETECTION 


Example: 


PRIORITIZING 


Example: 


SHORT-TERM 

MEMORY 


structure  and  meaning  to  the  information; 
Generalization  of  working  hypotheses  from 
specific  events; 

Discerning  basic  differences  and 
relationships  among  symbols,  figures 
and  figure  patterns;  Generating  a  new 
solution  to  a  problem;  Ability  to  make 
a  knowledgeable  assumption  using  incomplete 
data. 

Checking  the  adequacy  of  a  proposed  aircraft 
maneuver. 

Mental  storage  of  knowledge  over  a 
period  of  time  and  selective  recall 
of  items  relevant  to  a  situation. 

Remembering  aircraft  characteristics;  Remembering 
procedural  instructions  or  letters  of  agreement 
relevant  to  an  uncommon  situation,  such 
as  an  airshow  or  large  flight  formation. 

Translation  of  uncertainty  into 
probability;  Assigning  a  subjective 
probability  regarding  the  likelihood  of 
an  event  occurring;  Ability  to  use 
probabilities  to  identify  optimal 
courses  of  action. 

Assessing  the  risk  of  an  aircraft  maneuver. 

Recognition  of  the  physical  movement 
of  a  visual  object;  Estimation  of  its 
direction  or  speed. 

Observing  aircraft  on  the  Situation  Display  responding 
to  a  clearance  or  advisory. 

Ordering  of  events  in  sequence; 

Establishing  priorities. 

Deferring  a  request  for  flight  plan  changes  in  the 
presence  of  more  urgent  activity. 

Mental  storage  and  selective  recall  of 
of  relevant  information  over  a  brief 
period  of  time. 


Example: 

SPATIAL 

SCANNING 

Example: 

VERBAL 

FILTERING 

VISUALIZATION 

Example: 


Briefly  retaining  and  entering 
an  aircraft  call  sign. 

Rapid  identification  or  detection  of 
objects  or  events  displayed  in  a  wide 
or  complicated  visual  field. 

Observing  the  Situation  Display  for  new  aircraft; 
Searching  for  data  in  a  table. 

Same  as  FILTERING,  but  limited  to  voice 
communications. 

Observation  of  spatial  patterns  and 
subsequent  mental  transformations  into 
other  spatial  patterns. 

Determining  the  effect  of  a  proposed  aircraft 
maneuver  on  other  aircraft;  Comparing  intended 
time-position  profiles  for  intersection  in 
position/altitude/time. 
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Applicant  OPM  ATCS  battery  performance 

Mean  Scores  on  OPM  ATCS  battery  component  tests  for 
OPM  open  period  from  April-October  1985 
N=8,826 


Measure  Mean  Std.  Dev. 

Multiplex  Controller 

Aptitude  Test  (MCAT)  73.0  16.6 

Abstract  Reasoning  (ABSR)  31.5  9.3 

Occupational  Knowledge 

Test  (OKT)  29.2  1 1 .6 

Transmuted  Composite  75.5  12.5 

(TMC) 

OPM  Rating  (RAT)  76.4  13.3 
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Academy  Entrant  OPM  ATCS  battery  performance 

Mean  Scores  on  OPM  ATCS  battery  component  tests 
for  October  1985  through  September  1986  Academy  Entrants 


N=402 

Measure 

Mean 

Std.  Dev. 

Multiplex  Controller 

Aptitude  Test 

94.3 

5.8 

Abstract  Reasoning 

40.6 

5.5 

Occupational  Knowledge 

Test 

36.5 

14.6 

Transmuted  Composite 

91.8 

4.3 

OPM  Rating 

93.7 

4.0 
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Academy  Performance  Measures 

Mean  Scores  for  Academy  Performance  Measures 
October  1985  through  September  1986  Competitive  Entrants 

N=402 


Block  Test  Ave. 
Controller  Course 

95.7 

4.4 

Test 

Ave.  5  of  6  Laboratory 

93.4 

5.4 

Problems 

Ave.  Instructor 

74.0 

7.1 

Assessment 

Ave.  Technical 

84.6 

5.5 

Assessment 

Controller  Skills 

54.1 

10.8 

Test 

82.0 

7.5 

Comprehensive  Score 

79.7 

5.0 
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Correlations  between  Academy  and  OPM  battery  component  scores 
FY  1986  competitive  entrants  assigned  to  en  route  option 

N=402 


OPM  measures 


Measure 

MCAT 

ABSR 

OKT 

TMC 

RAT 

Academy  measures 

Block  Test  Ave. 

.02 

.03 

.14* 

.02 

.11 

Controller  Course 
Test 

.01 

.02 

.25* 

.02 

.19* 

Ave.  5  of  6  Lab. 
Problems 

.22* 

.04 

.00 

.20* 

.15* 

Ave.  Instructor 
Assessment 

.26* 

.05 

.03 

.25* 

.19* 

Ave.  Technical 
Assessment 

.23* 

.05 

.01 

.22* 

.17* 

Controller  Skills 
Test 

.16* 

.12* 

.03 

.19* 

.18* 

Comprehensive 

Score 

.24* 

.08 

.05 

.23 

.21* 

‘Correlations  were  significantly  different  from  0  at  pc.OI. 
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